Simulating Ideal and Average Users

We propose a framework for deterministic simulation of user behavior that allows to analyze the cost-gain-based performance on single result lists or whole search sessions. The ideal user representing optimal behavior (i.e., most gain with lowest effort) is contrasted with more “average” users that employ the spreading activation model from cognitive theory. On TREC Session Track data, the ideal user achieves about double the gain of real users at the same costs while the average gain of our different simulated users correlates well with the sessionDCG metric—another argument for that metric in session-based evaluation.